Seventh Practice ML¶
In this practice, we will learn Regularizations and how to use Ensembles to get predictions from more than one model.
We will also learn how to perform Hyper-Parameters Search and how to use the models KNN and LWLR.
We will learn how to get code from other developers from PyPi or GitHub and use it in our work.
We will also learn how to use AutoViz to show fast DataFrame graph reports.
PyPi¶
The Python Package Index (PyPI) is a repository of software for the Python programming language.
PyPI helps you find and install software developed and shared by the Python community.
Package authors use PyPI to distribute their software.
GitHub¶
GitHub is a code hosting platform for version control and collaboration.
It lets you and others work together on projects from anywhere.
It is also a common way to share the code you wrote with other developers.
Sometimes, the regular packages we are using are not enough, and we want to use things that are not officially implemented yet.
We can write them down ourselves, or search for implementations written by other developers.
These implementations will mostly be hosted at GitHub.
If we want to use them and their developers did not upload them to PyPi (to be downloaded easily with pip), we need to download them directly from their repositories in GitHub.
We will use git clone to clone repositories into our machine.
Downloads, Imports, and Definitions¶
We update packages that their Colab version is too old.
!pip install --upgrade plotly
!pip install autoviz
We import our regular packages.
# import numpy, matplotlib, etc.
import numpy as np
import pandas as pd
import seaborn as sns
import plotly.express as px
import matplotlib.pyplot as plt
import plotly.graph_objects as go
# sklearn imports
from sklearn import metrics
from sklearn import pipeline
from sklearn import linear_model
from sklearn import preprocessing
from sklearn import neural_network
from sklearn import model_selection
from sklearn.pipeline import Pipeline
from sklearn.model_selection import KFold
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import LeavePOut
from sklearn.linear_model import SGDRegressor
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import OrdinalEncoder
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
Data Exploration¶
We use the Vinho Verde's White Wines dataset.
Explanation¶
Source:
P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis.
Modeling wine preferences by data mining from physicochemical properties.
In Decision Support Systems, Elsevier, 47(4):547-553. ISSN: 0167-9236.
In the above reference, two datasets were created, using red and white wine samples.
The inputs include objective tests (e.g. PH values) and the output is based on sensory data (median of at least 3 evaluations made by wine experts).
Each expert graded the wine quality between 0 (very bad) and 10 (very excellent).
Relevant Information:
The dataset is related to white variants of the Portuguese "Vinho Verde" wine.
Only physicochemical (inputs) and sensory (the output) variables are available (e.g. there is no data about grape types, wine brand, wine selling price, etc.).
These datasets can be viewed as a classification or regression task.
The classes are ordered and not balanced (e.g. there are many more normal wines than excellent or poor ones).
white wine samples - 4898.
Number of Attributes - 11 + output attribute
Attribute information¶
Input variables (based on physicochemical tests):
- fixed acidity (tartaric acid - g / dm^3) - most acids involved with wine or fixed or nonvolatile (do not evaporate readily).
- volatile acidity (acetic acid - g / dm^3) - the amount of acetic acid in wine, which at too high of levels can lead to an unpleasant, vinegar taste.
- citric acid (g / dm^3) - found in small quantities, citric acid can add 'freshness' and flavor to wines.
- residual sugar (g / dm^3) - the amount of sugar remaining after fermentation stops, it's rare to find wines with less than 1 gram/liter, and wines with greater than 45 grams/liter are considered sweet.
- chlorides (sodium chloride - g / dm^3) - the amount of salt in the wine.
- free sulfur dioxide (mg / dm^3) - the free form of SO2 exists in equilibrium between molecular SO2 (as a dissolved gas) and bisulfite ion; it prevents microbial growth and the oxidation of the wine.
- total sulfur dioxide (mg / dm^3) - the amount of free and bound forms of S02; in low concentrations, SO2 is mostly undetectable in wine, but at free SO2 concentrations over 50 ppm, SO2 becomes evident in the nose and taste of wine.
- density (g / cm^3) - the density of water is close to that of water depending on the percent alcohol and sugar content.
- pH - describes how acidic or basic a wine is on a scale from 0 (very acidic) to 14 (very basic); most wines are between 3-4 on the pH scale.
- sulphates (potassium sulphate - g / dm3) - a wine additive that can contribute to sulfur dioxide gas (S02) levels, which acts as an antimicrobial and antioxidant.
- alcohol (% by volume) - the percent alcohol content of the wine.
Output variable (based on sensory data):
- quality (score between 0 and 10)
Let's download the dataset from Github and explore it with Pandas tools.
# download whitewines.csv file from Github
!wget https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/whitewines.csv
# load the whitewines csv file
whitewines_df = pd.read_csv('/content/whitewines.csv')
whitewines_df
# show whitewines_df info
whitewines_df.info()
# show whitewines_df description
whitewines_df.describe()
We can also use autoviz to show a report on the data.
This report is based on graphs other than text.
# import autoviz and show report on usedcars_df
from autoviz.AutoViz_Class import AutoViz_Class
AV = AutoViz_Class()
dft = AV.AutoViz("", depVar='quality', dfte=whitewines_df, verbose=1)
# divide the data to features and target
t = whitewines_df['quality'].copy()
X = whitewines_df.drop(['quality'], axis=1)
print('t')
display(t)
print()
print('X')
display(X)
Let's see how many different scores we have in the target.
t.unique()
We can see that we have only 7 different scores (ordinal target).
We can try to use regression models and classification models on this dataset.
Regularization¶
When we have a case of High-Variance, we can use Regularizations, to reduce it.
With regularization, we can control the size of the model weights and thus control the variance.
We control the size of the model weights by adding a term to the loss function.
This term is called a penalty to high weights.
We have learned about three regularization techniques:
- L1 (also called Lasso).
- L2 (also called Ridge).
- Elastic Net (a combination of Lasso and Ridge).
L1 - Lasso¶
The penalty is fixed for every weight.
It is going up or down with fixed-size steps.
The unnecessary weights tend to get to zero (it is practically feature selection).
L2 - Ridge¶
When the weights are big, this penalty adds big numbers to the loss function and thus reducing the weights by a lot.
When the weights are small, this penalty adds small numbers to the loss function and thus reducing the weights a little bit.
The weights that are unnecessary still stays in the model (but with small values).
ElasticNet¶
Combination of L1 (Lasso) and L2 (Ridge).
We can use these regularizations as Hyper-Parameters in Scikit-learn SGDRegressor and SGDClassifier.
Wen can also use regular GD with Scikit-learn Lasso, Ridge and ElasticNet.
Let's start with SGDRegressor.
# print lasso, ridge and elasticnet scores as regression
from sklearn.model_selection import cross_val_score
sgd_lasso = make_pipeline(StandardScaler(), SGDRegressor(penalty='l1', random_state=1))
sgd_ridge = make_pipeline(StandardScaler(), SGDRegressor(penalty='l2', random_state=1))
sgd_elastic = make_pipeline(StandardScaler(), SGDRegressor(penalty='elasticnet', random_state=1))
print('sgd_lasso', cross_val_score(make_pipeline(StandardScaler(), sgd_lasso), X, t, cv=15).mean())
print('sgd_ridge', cross_val_score(make_pipeline(StandardScaler(), sgd_ridge), X, t, cv=15).mean())
print('sgd_elastic', cross_val_score(make_pipeline(StandardScaler(), sgd_elastic), X, t, cv=15).mean())
Let's try the classification approach and use SGDClassifier.
# print lasso, ridge and elasticnet scores as classification
sgd_lasso = make_pipeline(StandardScaler(), SGDClassifier(penalty='l1', random_state=1))
sgd_ridge = make_pipeline(StandardScaler(), SGDClassifier(penalty='l2', random_state=1))
sgd_elastic = make_pipeline(StandardScaler(), SGDClassifier(penalty='elasticnet', random_state=1))
print('sgd_lasso', cross_val_score(make_pipeline(StandardScaler(), sgd_lasso), X, t, cv=15).mean())
print('sgd_ridge', cross_val_score(make_pipeline(StandardScaler(), sgd_ridge), X, t, cv=15).mean())
print('sgd_elastic', cross_val_score(make_pipeline(StandardScaler(), sgd_elastic), X, t, cv=15).mean())
We can see that the classifiers predicted better than the regressors.
We can try and create ordinal classifiers that will use the fact that there is an order to the labels.
We will not do it here - try it yourself.
Hyper-Parameters Search¶
Most of our models have a lot of parameters that can be adjusted.
Each parameter value can make our model better (or worse).
We want to be able to find the best hyperparameters for our models.
We have two approaches:
- Grid Search
- Random Search
Grid Search¶
When we want to check every parameter possible, we will use Grid Search.
We will try all combinations of parameters and find the best one, that gives us the best score.
This may be a little exhaustive, especially when we want to check a lot of parameters and values.
Random Search¶
We can choose to get random combinations of parameters and check the score on them.
This will not be as accurate as Grid Search, but it will take less time.
Let's use Scikit-learn GridSearchCV.
# train with grid search and get best parameters
from sklearn.model_selection import GridSearchCV
X_normalized = StandardScaler().fit_transform(X)
hyper_parameters = {'penalty': ('l2', 'l1', 'elasticnet'), 'alpha':[0.0001, 0.001, 0.01, 0.1]}
gs_model = GridSearchCV(SGDClassifier(random_state=1), hyper_parameters).fit(X_normalized, t)
print('gs_model', gs_model.best_score_)
print('best params', gs_model.best_params_)
We can see that the best parameters on this model (obtained with Grid Search) were penalty=elasticnet and alpha=0.001.
It may change with a different random_state.
Now let's try Scikit-learn RandomizedSearchCV.
We will use Scipy stats.uniform to get uniformly randomize values of alpha.
We will use Numpy random.seed to make sure that we get the same result each time we run this cell.
# train with random search and get best parameters
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform
np.random.seed(1)
distributions = dict(alpha=uniform(loc=0, scale=1), penalty=['l2', 'l1', 'elasticnet'])
rs_model = RandomizedSearchCV(SGDClassifier(), distributions, random_state=1).fit(X_normalized, t)
print('rs_model', rs_model.best_score_)
print('best params', rs_model.best_params_)
The best parameters on this model (obtained with Random Search) were penalty=l2 and alpha=0.417022004702574.
The R2 score is a little less than the score of the Grid Search, but it may change with a different random seed.
Ensembles¶
We can use a collection of models to make more accurate predictions and lower the variance.
If we use regression, we can take the mean of all the predictions of the models.
If we use classification, we can take the mean of all the probabilities of the model or choose the class that most of the models chose for some sample.
It is like "The wisdom of the crowd".
One model may be wrong, but a lot of different models are less prone to errors.
We are going to use two types of ensembles:
- Bagging (with NFold or with Bootstrap).
- Boosting.
Bagging¶
We create a few bags of samples from the original dataset.
We train a model on each of the bags of samples, and we return the combined score.
The bags can be created using NFold (same as KFold CV).
we simply divide the data into N parts and use KFold CV (where K=N).
We save all the N (or K) models and use them as an ensemble.
The bags can be created using Bootstrap.
We draw samples out of the dataset (with replacement) and train the model on each group of samples.
We save all the models and use them as an ensemble.
Boosting¶
We create a model and train it on the data.
We take the samples that the model predicted incorrectly and multiply them (thus giving them more weight in the next training).
We do this until we have few models, each of them is an expert on some type of samples.
We combine all the model's predictions and return a combined score.
There are few boosting algorithems (AdaBoost, GradientBoost, etc.).
Let's start with Scikit-learn BaggingClassifier.
When we use it with bootstrap=False we will address it as NFold, and when we use it with bootstrap=True we will address it as Bootstrap.
Let's start with NFold Bagging.
# get score with nfold bagging
from sklearn.ensemble import BaggingClassifier
bag_fold_model = BaggingClassifier(base_estimator=SGDClassifier(), n_estimators=20, random_state=1, bootstrap=False).fit(X_normalized, t)
print('bag_fold_model', bag_fold_model.score(X_normalized, t).mean())
Let's try the Bootstrap Bagging.
# get score with bootstrap bagging
bag_boot_model = BaggingClassifier(base_estimator=SGDClassifier(), n_estimators=20, random_state=1, bootstrap=True).fit(X_normalized, t)
print('bag_boot_model', bag_boot_model.score(X_normalized, t).mean())
In our case, NFold Bagging got better results but the difference is small and it may change if we use bigger n_estimators.
Let's try AdaBoosting.
# get score with ada boosting
from sklearn.ensemble import AdaBoostClassifier
ada_boost_model = AdaBoostClassifier(n_estimators=100, random_state=1).fit(X_normalized, t)
print('ada_boost_model', ada_boost_model.score(X_normalized, t).mean())
In our case, the bagging ensemble performs best.
LWLR¶
We can use a special form of Linear Regression.
This form is called Locally Weighted Linear Regression.
This is the equation:
We can see that we added a weight (beta) for every sample.
This weight can help us emphasize the importance of the samples that are similar to our test sample.
For every test sample, we train the model from scratch and give each training sample a weight that is corresponding to the distance from the test sample.
We can use the Gaussian function as weight:
If tao is small, only closer samples are taking into consideration in the WMSE (big distances get small weight (beta)).
If tao is big, we get our regular MSE (big distances affect as much as small distances).
Scikit-learn does not have LWLR in its arsenal, so we need to use a packege from GitHub.
This package is not stored in PyPi, so we can not download it with pip install.
We need to use git clone .
# clone the lwlr repo from github
!git clone https://github.com/qiaochen/CourseExercises
Let's check the LWLR model with k=1.
# get cv score for lwlr with k=1
from CourseExercises.lwlr import LWLR
arr_X_normalized = np.asarray(X_normalized)
cross_val_score(LWLR(k=1), arr_X_normalized, t, cv=5, scoring='r2').mean()
Let's get the best k for the LWLR model.
It may take some time (we are building the model from scratch for every test sample), so let's check how long it takes.
%%time
# get best k for lwlr (show the calculation of this sell)
hyper_parameters = {'k': list(range(1, 10))}
gs_lw_model = GridSearchCV(LWLR(k=1), hyper_parameters, scoring='r2').fit(arr_X_normalized, t)
print('gs_lw_model', gs_lw_model.best_score_)
print('best params', gs_lw_model.best_params_)
KNN¶
We can take the idea we saw in LWLR to the extreme and create a model that predicts only based on the closest training samples to a test sample.
This model is called K Nearest Neighbors.
We can choose the k and the model will calculate the prediction for each test sample, based on the closest k training samples to the test sample.
We need to determine what is the meaning of close.
We need to use some sort of distance function to determine the closeness of each training sample to the test samples.
We can use the Euclidean distance:
We can use KNN in classification tasks or regression tasks.
When we use it in a regression task, we can take the mean of the target of all the neighbors.
When we use it in a classification task, we can take the mean of the probability of all the neighbors, or we can use voting, and choose the label that got the most votes from the neighbors.
We can choose to give all the neighbors that are in the decision group, the same weight in the vote, or we can give the closest neighbors higher weight than the farthest.
Let's use Scikit-learn KNeighborsClassifier.
from sklearn.neighbors import KNeighborsClassifier
hyper_parameters = {'n_neighbors': list(range(1, 20))}
gs_neigh_model = GridSearchCV(KNeighborsClassifier(n_neighbors=5), hyper_parameters).fit(arr_X_normalized, t)
print('gs_neigh_model', gs_neigh_model.best_score_)
print('best params', gs_neigh_model.best_params_)
We can see that the best n_neighbors is 17.
let's try the Scikit-learn KNeighborsRegressor.
from sklearn.neighbors import KNeighborsRegressor
hyper_parameters = {'n_neighbors': list(range(1, 20))}
gs_neigh_model = GridSearchCV(KNeighborsRegressor(n_neighbors=5, weights='distance'), hyper_parameters).fit(arr_X_normalized, t)
print('gs_neigh_model', gs_neigh_model.best_score_)
print('best params', gs_neigh_model.best_params_)
As we expected, it did worse than the KNeighborsClassifier on this dataset.
More Information¶
Guide on how to upload python packeges to PyPi:
How to upload your python package to PyPi
Explanation of few EDA libraries in python:
4 Libraries that can perform EDA in one line of python code
Explanation of Vinho Verde wines:
Portuguese Vinho Verde wine: everything you need to know
Kaggle notebook on the white wine dataset:
KNN for classifying wine quality
Kaggle notebook on the white wine dataset:
Predicting White Wine Quality
Explanation on Lowess smoothing:
Lowess Smoothing: Overview
Explanation on regularization:
REGULARIZATION: An important concept in Machine Learning
Article about the geometry of Ridge and Lasso regularizations:
Regularization and Geometry
Explanation on Ridge, Lasso, and Elastic Net regularizations:
An Introduction to Ridge, Lasso, and Elastic Net Regression
Explanation of the differences between Ridge and Lasso regularizations:
Intuitive and Visual Explanation on the differences between L1 and L2 regularization
Guide on how to use regular classifier as an ordinal classifier:
Simple Trick to Train an Ordinal Regression with any Classifier
A list of predefined scores for Sciking-learn:
Common cases: predefined scores
Explanation on LWLR and a code sample:
Linear Regression: How to overcome underfitting with Locally Weighted Linear Regression (LWLR)
Example of LWLR in python:
Locally Weighted Linear Regression in Python
Short Explanation on LWLR:
ML | Locally weighted Linear Regression